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ABSTRACT 

The purpose of this paper is to explore the effectiveness of Action Design Research (ADR), a combination of Action 
Research and Design Science Research, as a methodology to examine how the implementation of e-leaming will affect 
the learning outcomes for staff training in an organisational context. The research involves an intervention in the finance 
course offerings for staff training in the workplace. The paper briefly introduces the research problem of an inflexible and 
inefficient finance training offering, and the proposed solution of implementing e-leaming. It then describes the seven 
principles of ADR in detail, in a tabular format, with particular reference to their application in this study. The paper then 
summarises how the seven principles fit into the four stages of ADR, namely, (1) problem formulation, (2) building, 
intervention and evaluation, (3) reflection and learning, and (4) Formalisation of learning. After discussing the ADR 
principles and how they relate to the research study, the paper concludes with the suitability and advantages of adopting 
an ADR approach to e-learning research. These benefits appear to be meeting the challenge of IS as an applied discipline, 
by implementing a solution in a real world situation, whilst also adding to e-leaming theory and academic knowledge. 
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1. INTRODUCTION 

In the light of the need to accommodate the continually changing business environment, the swift 
dissemination of relevant, up-to-date information is essential. When using an integrated financial system, 
changes in the system necessitate almost instantaneous communication, and are often accompanied by 
training issues. Apart from system changes, there is a basic requirement that the workforce have a working 
knowledge of the system, and the skill to extract relevant information from it. If as a consequence of 
insufficient or ineffective staff training, employees are not skilled or technically capable, then a major source 
of competitive advantage is lost, and the organisation’s workforce cannot realise its full potential as a source 
of intellectual capital. Additionally, there appears to be a need for more input from IS research into the 
practice of e-learning in the workplace, specifically its effectiveness and on-going usage. 

The immediate objective of this research is to convert the current instructor-led finance training courses 
for employees at an organisation to computer-based courses. The long-term objective is to create a learning 
environment that meets both the business need for a knowledgeable, skilled body of staff, as well as the 
individual adult learner’s need for meaningful, practical and flexible instruction (Figure 1). The current 
instructor-led training is rigid as the venue and the trainer need to be booked, and the training schedule is set 
up a year in advance. This causes problems for trainees, trainers and line-managers. Having established that 
there is a problem situation, the difficulty is to employ a research methodology that has a practical outcome 
for the organisation, whilst simultaneously meeting academic standards of credibility, dependability and 
transferability. 
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Training Intervention Research Summary 

• WHO? Specific members of the workforce that need access to financial system 

• HOW? Computer based training courses (e-learning) 

• WHY? Learn and/or upgrade technical skills & financial knowledge required to 
operate the financial system, as well as interpret financial reports (Business need) 
Meaningful, practical & flexible instruction (individual adult learner's need) 

• LONG TERM WHY? Increased competitive advantage for organisation 

Figure 1 . Proposed research to investigate the change from classroom based to computer-based training for compulsory 

finance training 

There has been debate as to the nature of Information Systems (IS) research, and where it situates itself, 
for some time (Lee, 2010; March & Niederman, 2012; Rosemann & Vessey, 2008; Walsham, 2012). Lee 
(2010) examined key concepts of IS research and concluded that the future of IS research may lie in the 
“sciences of the artificial”, as distinguished from the “sciences of the natural” by Simon (1969). The over- 
riding knowledge requisite for this type of discipline is “its efficiency and effectiveness for bringing into 
existence an artefact needed to solve a given problem, achieve a given goal, or otherwise fulfil a given need 
that is facing people in the real world” (Lee, 2010, p346). This is an opposite view to that of the sciences of 
the natural, which seek to theorise about things that are already there. As the objective of this research is to 
intervene in a real world setting, and to make changes based on a designed artefact, two research approaches 
were considered, namely Design Research (DR), also known as Design Science Research, and Action 
Research (AR). 

Arising from the debate regarding the IS discipline and how it should differentiate itself from other 
disciplines, such as Computer Science, it may be suggested that the IS discipline stands at the intersection of 
people, organisations and technology (Hevner et ak, 2004; Lee, 2010; March & Niederman, 2012). 
According to livari (2007), Design Research (DR) is problem focused and seeks to design an innovative 
product, or artefact, that addresses unsolved problems within an organisation. In the context of this paper, 
artefacts are defined as software and/or hardware that are assembled to form the object to be implemented. 
This “object” inscribes certain organisational structures into its form as it emerges, involving more than just 
technology in its conceptualisation and goals. Sein et al. (2011, p.38) describe artefacts as “ensembles 
emerging from design, use, and ongoing refinement in context”. There is a build and evaluate process which 
forms a loop which is usually iterated as the design is refined, before the final artefact is produced (Hevner et 
ak, 2004). DR must also contribute to research by making “clear and verifiable contributions in the areas of 
the design artefact, design foundations and/or design methodologies” (Hevner et ak, 2004, p83). However, a 
possible limitation of design science research is that the building of the artefact is separate from the 
evaluation step, and the value of the research lies in its ability to solve the original problem, rather than 
testing it in a real life setting (Baskerville et ak, 2009; Sein et ak, 2011). The intervention or introduction of 
the artefact into the organisation is a secondary factor for much design science research, as the primary factor 
concerns the actual design of the artefact and testing is often not taken out of the laboratory (Cole et ak, 
2005). 

According to Baskerville & Wood-Harper (1996, p.243), “Action research is regarded by many as the 
ideal post-positivist social scientific research method for IS research”. They base this statement on the 
premise that IS, being a highly applied field, requires integration with the psychological and social facets of 
changing or introducing a technology into the workplace. People are directly affected by these changes, and 
both monitoring and evaluating their reactions should perhaps be considered an important part of the 
research. It seems that acting on feedback or evaluations from end-users is an important part of a successful 
implementation, and action research would appear to be ideally suited to allow for this. In order to increase 
user acceptance of the online courses, it is felt that feeding user evaluations and comments directly back into 
the course design would be advantageous for both the trainees and the organisation. Action research involves 
a strategy of formulating theory (Baskerville & Pries-Heje, 1999) and the researcher not only intervenes but 
participates in the study, which supports an interpretive, anti-positivist philosophy. Action research is 
iterative in nature, and the implemented process or object is refined during the cycles. It also includes what 
McKay and Marshall (2001) refer to as the duality of action research, i.e. the researcher is both participant 
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and observer. The researcher is required to keep both roles in mind, and separated, as the study progresses. 
On the one hand, the researcher is acting as the manager of the intervention, and as such is directly involved 
in the intervention, whilst at the same time, the researcher is required to reflect on the process in order to 
answer the research questions and formulate theories or themes. According to McNiff and Whitehead (2006, 
pi 3), “The purpose of action research is to generate living theories about how learning has improved practice 
and is informing new practices”. However, action research has frequently been criticised for the occurrence 
of personal bias or over-involvement of the researcher, lack of rigour, and has sometimes been labelled as 
consulting rather than researching (Baskerville & Wood-Harper, 1996). 

The purpose of this paper is to attempt to address the reported limitations of DR and AR research 
approaches, by adopting the research method proposed by Sein et al. (2011). It is hoped that Action Design 
Research (ADR) will be effective in addressing an actual problematic situation in an organisational setting. 
The research encompasses the building of an innovative Information Technology (IT) artefact, whilst 
allowing feedback from the users and the organisation to influence the construction thereof in reiterative 
cycles, allowing both learning from the intervention, and simultaneously producing academic theory. 

This paper may be of interest to any researchers seeking a research approach that meets the need for 
relevance and immediacy in an organisational setting, whilst also contributing to empirical research on the 
effectiveness or otherwise of e-learning, and adding to the body of knowledge regarding theories of 
organisational learning and training. The need for this type of IS research is endorsed by Baskerville and 
Wood-Harper (1996) from the AR perspective, and Hevner et al. (2004) from that of the DR perspective, and 
Cole et al. (2005), who investigated the use of both research methodologies. 


2. ACTION DESIGN RESEARCH 

Cole et al. (2005) feel it would be proactive of IS researchers to consider using both design science research 
and action research together in order to achieve a rigorously designed artefact that is evaluated in a real life 
organisational context to solve or to ameliorate a perceived problem within that organisation. ADR, as 
described by Sein et al. (2011, p.40), “is a research method for generating prescriptive design knowledge 
through building and evaluating ensemble IT artefacts in an organizational setting”. It has been designed to 
address the challenge of assisting IS practitioners by intervening in real world situations, whilst also building 
theory that is academically rigorous. This also serves to answer the call of making IS research relevant to 
practitioners and other IS professionals, and “promoting engaged scholarship through action and design” 
(Conboy et al., 2012, pi 14). It seeks to overcome the perceived limitations of design science research (DR) 
and action research (AR), as well as addressing livari’s (2007) concerns regarding the differences between 
the two methods. By placing the IT artefact at the centre, this methodology also addresses Orlikowski and 
lacono’s (2001, p.l30) call to “increase attention and explicit consideration of IT artifacts in all (IS) studies”. 
ADR has strict, explicit principles which are sometimes lacking in AR, whilst the iterations and simultaneous 
building, intervention and evaluation address the sequencing difficulties of DR, when attempting to use the 
designed artefacts in organisational settings. By not taking the setting or context of the organisation into 
account, whilst designing the artefact, DR does not usually capture its emergent nature. There is 
interdependence between design and use in the organisation, which needs to be captured or inscribed into the 
artefact. This interdependence is highlighted by ADR. 

There are four stages in the ADR method. These are (1) problem formulation; (2) building, intervention 
and evaluation; (3) reflection and learning, and finally, (4) formalisation of learning. It is important to note 
that the four stages are not engaged within a step or waterfall sequence, but are cyclical and reiterative. Sein 
et al. (2011) also describe seven principles of ADR. These principles are detailed in Table 1 below, with 
particular application to this research. 
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Table 1. The seven principles of ADR, as applied to this research 



Principle 

Description (Sein et al., 2011) 

Application 

1 

Practice- 

inspired 

research 

The problem is drawn from the 
real-world and is used to create 
knowledge regarding a class of 
problems, which are typifled by 
this particular problem 

Real world problem Need to accommodate continually changing business 
environment by swift dissemination of relevant, up-to-date information. 

Class of problem - effective, efficient use of e-leaming as a tool that is 
accepted by both the business and the trainees. 

This problem - converting current instructor- led training to online training that 
continues to meet business need for knowledgeable, skilled workforce, whilst 
accommodating trainee’s need for practical and flexible instmction. 

2 

Theory- 

ingrained 

artefact 

The original desigji of the artefact 
is based on existing theories and 
technologies, but the design may 
change once it is evaluated within 
the organisation. 

Extensive literature review of current e-leaming body of knowledge resulting in 
an artefact (the online training module) that enconpasses many success fectors 
identified, and allows for learner feedback and evaluation. Review covers 
system design, system delivery and system outcome; learning approaches and 
strategies. 

3 

Reciprocal 

shaping 

This is where the IT artefact and 
the organisational domain influence 
each other, causing an iterative 
cycle to be set in motion. The 
design of the artefact will change 
as it is used in the organisational 
setting, and the organisation may 
be changed by the use of the 
artefrct. 

A^ha version of artefact tested by other trainers, changes to module made 
based on feedback. Beta version launched in controlled conditions (le. in lab, 
with instmctor present, but all other conditions of online environment met - 
duration, test taken at learner’s discretion). Further changes made to module 
based on learner feedback. Appropriately trained, motivated and enabled 
workforce has positive effect on organisation’s financial reporting, leading to 
more modules, with targeted objectives, being offered online. 

4 

Mutually 

influential 

roles 

This refers to how the multiple 
participants in the ADR project 
share their specialised knowledge 
and learn from each other. The 
researchers bring in the theory and 
the practitioners bring in the work 
practices. 

Researcher designs course module using knowledge of best practices gained 
from literature on e-leaming, with input from practitioners regarding course 
content, and specific working environment. With reiterative cycles, workforce 
input also influences shaping of module. 

5 

Authentic 

and 

concurrent 

evaluation 

This means that the process of 
evaluation is never separated from 
the building and intervention, but is 
an integral part of both. Depending 
on the form of the artefact, 
evaluation can be either formative 
or summative, but must always be 
allowed to occur spontaneously in 
the organisational context, rather 
than in a controlled setting. 

The initial course module is built and offered to trainees, in working 
environment. Feedback and comments are discussed by researcher and 
trainers, and changes made to artefact as appropriate. Unexpected 
consequences are immediately visible, and can be acted upon promptly. The 
artefect is continuously being evaluated and inplemented in on-going wave in 
dynamic environment. 

6 

Guided 

emergence 

This highlights the interaction 
between the initial design and how 
its implementation in the 
organisational setting causes a 
continuous evolvement and re- 
working of the emerging artefect. 

The artefact Is rolled out into the real life situation, and feedback from 
participants and assisting practitioners is immediately acted on by course 
designer/researcher, with re-designed module being offered almost 
immediately. Feedback will arise from course content and organisational 
environment, le. equipment, setting, timing, control Each cycle will result in a 
re-working of the artefact (course module) as required by the evaluations. 

7 

Generalized 

outcomes 

This ensures that the learning that 
has occurred during the iterations 
is developed so that the solution of 
the specific problem can be 
generalised to a class of problems. 

The researcher will describe the learning and re-desigji of arteiact based on 
user feedback and practitioner evaluation. This learning will in turn be linked 
back to current literature on best practices of e-leaming in organisational 
environment. 
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The seven principles, detailed in Table 1, are aligned with the ADR method stages, and should be realised 
during the relevant stage. Thus, the first stage of problem formulation is addressed by the principles of 
“practice-inspired research” and “theory-ingrained artefact”, the second stage of building, intervention and 
evaluation is addressed by the principles of “reciprocal shaping”, “mutually influential roles” and “authentic 
and concurrent evaluation”. The third stage of reflection and learning is addressed by the principle of “guided 
emergence”, and the fourth stage, formalization of learning, is addressed by the principle of “generalized 
outcomes” (Figure 2, Sein et al., 201 1). 


3. Reflection 

and Learning 


Principle 6. Guided 
Emergence 


4. Formalization of 
Learning 


Pnrcpte 7: Generalized Outcomes 



Figure 2. The ADR method: Stages and principles (Sein et al., 2011) 

[Figure 2 from M. K. Sein, O. Henfridsson, S. Purao, M. Rossi, and R. Lindgren, “Action Design Research,” MIS 
Quarterly (35:1), 2011, p. 41. Copyright © 2011, Regents of the University of Minnesota. Reprinted by permission] 

The perceived problem which is to be investigated can arise from many sources within the organisation, 
such as practitioners, end-users, and the existing technology. In this research, the problems with the current 
classroom based training have been remarked on by trainees, trainers, line-managers and financial managers. 
To summarise the four stages, with the inherent principle(s), the first stage of ADR, “problem formulation”, 
requires that the researcher identifies and conceptualises the research opportunity; formulates initial research 
questions; casts the problem as an instance of a class of problems; identifies contributing theoretical bases 
and prior technological advances; secures long-term organisational commitment; and sets up roles and 
responsibilities (Sein et al., 2011, p.41). The aim of the principle of practice-inspired research is that the 
researcher produces “knowledge that can be applied to the class of problems that the specific problem 
exemplifies” (p.40). The principle of a theory-ingrained artefact, which is also aligned to this stage, ensures 
that the artefact to be designed and tested within the organisation is based on current theory, or theories. 

These theories should assist with the problem formulation, the identification of possible solutions and the 
initially created artefact, which is then introduced into the organisation for evaluation, as part of the second 
stage, “building, intervention and evaluation” (BIE). This introduction of the artefact into the real world 
situation should result in further cycles of re-designing or re-shaping, and re-evaluation of the artefact. Where 
the IT artefact is the focus of the research, Sein et al., (2011) suggest that alpha versions of the emerging 
artefact are tested and evaluated by the practitioners. After this strongly participatory process, a beta version 
of the artefact is introduced to the end-users themselves, and again, the resulting evaluations refine and shape 
the artefact (Figure 3). 
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Figure 3. ADR generic schema for IT-Dominant Building, Intervention and Evaluation (Sein et al., 201 1) 

[Figure 3 from M. K. Sein, O. Henfridsson, S. Purao, M. Rossi, and R. Lindgren, “Action Design Research,” MIS 
Quarterly (35:1), 201 1, p. 42. Copyright © 2011, Regents of the University of Minnesota. Reprinted by permission] 

The principles of “reciprocal shaping”, “mutually influential roles” and “authentic and concurrent 
evaluation” are addressed during the BIE stage. ADR suggests that the influence of the IT artefact and the 
organisational context are mutual and inseparable. This may result in the original stance of the researcher 
being changed by the feedback from the organisation, and the ultimate version of the IT artefact quite 
different from the original version. Additionally, the researcher and the practitioner also influence each other. 
The different insights offered by these ADR team members into the creation of the artefact, mean that the end 
result should be more effective being a combination of both theory and practice. It should be noted that 
individuals in ADR can perform multiple roles, but that these roles are clearly identified and responsibilities 
assigned in Stage 1. It should be clearly visible from figure 3, the generic schema for IT-Dominant BIE, that 
ADR does not separate evaluation from building. It is hoped that any unanticipated consequences are 
surfaced during the evaluation of the alpha version, which then allows for refinement of the artefact, before 
the beta version is introduced to the end user. The decisions regarding the shape of the IT artefact and the 
intervention in the real organisational setting should be entwined with constant evaluation. Sein et al. (2011) 
emphasise that due to the emergent nature of the artefact, the setting is not controlled. It is their belief is that 
achieving authenticity in a natural setting is more important for ADR. The tasks in Stage 2 are therefore to 
(1) discover the initial know ledge -creation target, (2) select or customise the BIE form; (3) execute the BIE 
cycle(s); and (4) assess the need for additional cycles (Sein et al., 201 1, p43). 

Stage 3, “Reflection and learning”, is continuous, and runs alongside both Stages 1 and 2. This is where 
the contributions to theory arise, and as the artefact emerges, adjustments to the research process need to be 
made as the researcher’s understanding of it increases. The principle that is attached to this stage is “guided 
emergence”. Stage 3 tasks are (1) reflect on the design and redesign during the project; (2) evaluate 
adherence to principles, and (3) analyse the intervention results according to the state goals (Sein et al, 2011, 
p44). The last stage of ADR is the “Eormalisation of learning”. The effect of the artefact on the organisation 
should be described as formalised learning, and it is hoped that the changes that were made during the BIE 
stage, will now be able to be explicated in order to enhance the original theories that were used to create the 
initial alpha version of the artefact. During this stage, the principle of “generalised outcomes” should be 
applied. Sein et al. (2011, p44) suggest this implies a shift from “the specific and unique” to “generic and 
abstract”, and that three levels are involved, the “generalisation of the problem instance, the generalisation of 
the solution instance, and the derivation of the design principles from the design research outcomes”. The 
tasks that should be accomplished during this stage are (1) abstract the learning into concepts for a class of 
field problems; (2) share outcomes and assessment with practitioners; (3) articulate outcomes as design 
principles; (4) articulate learning in light of theories selected; and (5) formalise results for dissemination 
(Sein et al., 201 1, p45). 
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According to Sein et al. (2011), ADR has been developed with the ultimate goal of innovative design 
knowledge for the particular class of problems. The method acknowledges that the artefact being designed 
emerges from the meeting of IT and the organisation, and that any unanticipated consequences of 
implementing the artefact, can be addressed immediately, and to the benefit of both IT, from a theory 
perspective, and the organisation, from a practice perspective. It is felt that this approach, which combines 
AR with DR will be the best approach to realising the objectives of this research, as well as delivering 
research that is “efficacious for solving the practical problems at hand” (Lee, 2010, p345). The performance 
of the final versions of the implemented modules will be measured using the criteria of efficacy, efficiency 
and effectiveness (Figure 4), as defined by Checkland & Poulter (2006). 


MEASURES: 

• Efficacy: 

o Measure 1\ Trainees able to extract relevant financial reports & accurate inputs in financial 
system 

Too/: Extracted financial reports, interviews with line managers 
o Measure 2: Trainee satisfaction with training 
Tool: Evaluation form 

• Efficiency: 

Measure: number of freed trainer hours 

Tool: Report comparing classroom & e-learning trainer hours 

• Effectiveness: 

o Measure 1\ Improved Quarterly financial reporting submissions 
Tool: Report comparing quarterly financial reports 
o Measure 2: Improved financial job performance as rated by Financial Managers 
Tool: Interviews 

(For all 3, researcher diary, notes, minutes of meetings) 

Figure 4. Proposed research measures to investigate the efficacy, efficiency and effectiveness of the computer-based 

training for compulsory finance training 


3. CONCLUSION 

When implementing e-learning in the context of the workplace, it is the personal experiences of the 
participants that is important. Whether the implementation is a success or not will depend on the subjective 
reaction that the trainees have when exposed to this method of training. It would seem that a good approach 
to ensure that the users of the system are involved is to use their own feedback to adjust the modules so that 
the final experience is both enabling and positive. Having researched both Action Research and Design 
Research as possible research approaches that are suitable for e-learning research, it appears that both 
approaches have their critics and limitations. Therefore, using a combination approach, designated by Sein et 
al. (2011) as Action Design Research, would seem advantageous. It is felt that this would be the most 
suitable way of solving the specific training concern, whilst at the same time, attempting to contribute 
towards improving knowledge in respect of e-learning implementations in the workplace and how to measure 
the efficacy, efficiency and effectiveness of such training (Iversen et al., 2004). It is also hoped that this 
approach will address the IS issue of being an applied discipline which needs to consider assisting 
practitioners, as well as adding to theory about IT and organisations, and the human resources within the 
organisations that use the IT. 

As this study attempts to contribute to practice by intervening in a real world situation, the study is 
limited to the compulsory training of the workforce at a single institution. The training modules affected are 
run by the finance department systems and user support team. Therefore, it would be questionable to make 
statistical generalisation from the findings. However, according to Lee and Baskerville (2003, p230), “In 
interpretivism, a theory’s pertaining only to the setting where it was developed would not detract from its 
validity or scientific status”. Therefore, as this a qualitative study, the objective is to generalise from the 
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individual findings to a theory within a particular setting, rather than to generalise from the sample to the 
population (Conboy et ah, 2012; Lee & Baskerville, 2003, Merriam, 1988). This study is set within the 
workforce of a particular organisation that employs a sophisticated, integrated financial system. Whether the 
findings may be applied to other organisations, both in Africa and abroad, is uncertain due to the composition 
of the workforce and the nature of the system, but the descriptive and practical nature of the research may 
enable other researchers facing a similar problematic situation within other organisations to use the theory 
and findings as a base to develop additional theories, or to compare and contrast interpretations. 
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